A process and system for automatically applying color correction to a region-of-interest

ABSTRACT

A system and methods for creating user definable windows for applying image processing, particularly useful in a post-production color correction equipment. The system allows a user of an image processing system such as a scene by scene color corrector to define a color region of interest in an image for purposes of applying image processing only to the region of interest, with automatic tracking of that region of interest over a plurality of frames in a scene. The user defined window, comprising a closed polygon, is converted on a frame-by-frame basis to a matte that is keyed with image processing such as color correction values. The user defined windows can be edited and stored and recalled for use in other frames. The color region of interest is automatically tracked from a beginning frame to the ending frame of a scene, with a vector window defined for each frame, so as to adapt to and compensate for changes in shape, size, and geometry of the region of interest during the succession of frames in the scene.

This application is a divisional of application Ser. No. 09/282,913,filed Mar. 31, 1999, now U.S. Pat. No. ______, entitled AUTOMATICREGION-OF-INTEREST TRACKING FOR A COLOR CORRECTION SYSTEM, which isincorporated herein in its entirety by reference.

TECHNICAL FIELD

The present invention relates generally to image processing, and moreparticularly relates to systems and methods that allow a user of animage processing system such as a scene by scene color corrector todefine a color region of interest in an image for purposes of applyingimage processing only to the region of interest, with automatic trackingof that region of interest over a plurality of frames in a scene.

BACKGROUND OF THE INVENTION

In certain types of image processing systems, especially colorcorrection systems employed in post-production equipment, systemoperators observe an image on a video monitor, adjust color and otherparameters of the image until it is aesthetically satisfactory, storethe parameters in system memory, and apply the parameter corrections toa sequence of images forming a scene. Various types of image processingare often employed to create, enhance, compress, filter, or otherwisemodify characteristics of an image.

In a video signal color correction system, color corrections of motionpicture film and/or video tape are typically made on a scene-by-scenebasis. A “scene” is a sequential collection of images shot from the samecamera, having the same viewpoint, composed in a certain way, etc. Asystem operator or “colorist” views a selected exemplary image or framefrom a scene. The colorist then applies color corrections via a controlpanel to adjust video parameters such as hue, saturation, luminance,etc. of a frame image being viewed. The correction settings are storedin system memory.

After the colorist is satisfied with the adjustments he or she has madeto the selected frame, the correction system, which is typicallycomputer-controlled, applies the stored color corrections to each framein the scene one at a time. The color-corrected frames of the scene arethen recorded on film or videotape. The steps are repeated for otherscenes in the film or video tape, often with different correctionsettings stored for different scenes. This process creates acolor-corrected master film or video tape that reflects coloradjustments to all frames in all scenes.

Devices are known in the art for defining a region of an image forapplying one set of color corrections, with other regions of the imagereceiving another, separate set of color corrections. Such devices areuseful when it is desired to isolate a certain area of an image toreceive special image processing. For example, in color correcting asoft drink commercial it may be desirable to isolate the beverage canfrom the remainder of the image so as to make the colors of the beveragecan stand out from the rest of the image.

Certain prior art image processing computer software, e.g. ADOBEPHOTOSHOP™, manufactured by Adobe Systems Corporation, Mountain View,Calif., employs edge detection algorithms for detecting color regionsand creating paths or Bezier curves between points. Such edge detectionalgorithms rely upon color differences between regions to define thepaths and produce odd results if there are discontinuities in the edgesof a region, or if the colors defining the regions are not sufficientlydissimilar. The PHOTOSHOP™ software, for example, vectorizes an entirecolor region and provides no control over the number of points in theregion. There is no way to modify a defined region except by changingregion qualification parameters. For these and other reasons, suchsoftware is not suitable for real time applications such as a scene byscene color corrector.

A system for creating user definable windows for applying imageprocessing is described in U.S. patent application Ser. No. 08/912,662,filed Aug. 8, 1997 entitled “User Definable Windows for Selecting ImageProcessing Regions.” This application is owned by the same assignee asthe present invention. In this system, the luminance values of a keyimage are captured in a key frame buffer. The key image is edited in a“paint” mode to enhance the luminance values or eliminate regions of theimage, to define a region of interest. The key image is then convertedto a 1-bit image. The 1-bit image is vectorized to derive a user definedwindow. The user defined window, comprising a closed polygon, isconverted on a frame-by-frame basis to a matte that is keyed with imageprocessing such as color correction values. User defined windows can beedited and stored and recalled for use in other frames. A user definedwindow for a first frame of a scene can be transformed or morphed into auser defined window for a final frame of a scene, with the systemautomatically interpolating between the beginning window and the endingwindow so that the window is automatically moved, resized, and changedin geometry during the succession of frames in a scene.

Although this system provides remarkable improvements in isolatingregions for receiving color correction in a scene-by-scene colorcorrector, only the luminance of the target image is employed to isolatea region of interest. Moreover, the system as described in this patentis not capable of automatically tracking a region of interest based onits color parameters (hue, saturation, and/or luminance) over a numberof frames—the system therein described “morphs” the polygon from a firstframe to a final frame of a scene.

There is a need, however, for a system that isolates a region ofinterest by its color parameters, e.g. hue, saturation, and/orluminance, or red/green/blue values, and employs those color parametersto establish and track the region of interest over number of frames in ascene, to effect automatic tracking of the region of interest over aplurality of frames.

SUMMARY OF INVENTION

The present invention is particularly adapted for use in connection withthe system described in U.S. patent application Ser. No. 08/912,662,filed Aug. 8, 1997, entitled “User Definable Windows for Selecting ImageProcessing Regions,” the disclosure of which is incorporated herein byreference and made a part hereof.

The system in this patent employs the concept of a key well known in thetelevision arts to define regions for receiving image processing. A goodexample of the use of a key is that of a television weathermansuperimposed on a map. The weatherman is typically shot with a videocamera standing in front of a solid blue color background. Otherequipment automatically generates the video weather map. Imageprocessing equipment is readily able to discern the outline of theweatherman relative to the blue background by use of color values. Theimage of the weatherman, which is called a “key”, is overlaid on the mapby essentially providing transparency for the blue regions of the keyoutside the weatherman's body. This allows the weather map to showthrough in the blue areas, and superimposes the weatherman on top of themap.

Related principles are employed in the present invention for defining auser defined key corresponding with a predetermined region of an imageand utilizing the key to derive a user defined window. Image processingsuch as a set of color corrections is applied within the region definedby the key-derived user defined window. A separate second type of imageprocessing, for example a second set of color corrections, is applied inregions outside the user defined window.

Briefly described, the present invention comprises a system and methodsthat allow creation of user definable windows for selecting imageprocessing regions. These windows can vary in geometry, size, andlocation during a number of frames in a scene. The invention isparticularly useful for allowing a colorist in a video image colorcorrection system to define windows for selecting regions for colorcorrection. The user typically starts by capturing an exemplary image orframe of a scene for use as a “key image” or “key frame”. The key imageis manipulated by adjusting parameters such as gain, clip, and limit toderive one or more key regions of the image to serve as keys. The keyregions are vectorized and stored in a computer system's memory inassociation with the key frame. The vectorized keys, which are thencalled “user defined windows”, are then used during a plurality offrames for defining regions to receive image processing such as colorcorrection, data compression, noise reduction, etc.

For use in a scene having a number of frames, the user defines a windowfor a first frame of the scene and a window for the last frame of thescene. The user may use the last frame as a key image and derive anotherkey for defining the final window, can use and modify the window for thefirst frame for defining the final window, or can use modify any otherpreviously defined window.

The system described in the referenced patent then interpolates or“morphs” the window of the first frame of the scene into the window ofthe final frame of the scene by linear interpolation across each frameof the scene. The windows are generated on a real time, frame by framebasis so that the color correction or other image processing can beviewed by the colorist on a video monitor. The windows thus changeduring the sequences of frames in the scene to compensate for changes inthe size, geometry, and position of the area of interest in the scene.

In a system constructed in accordance with the present invention, theoperator uses a pointer to select a point or group of points (a region)having desired color properties. The system automatically calculates themean of the red, green, and blue values of each pixel of the selectedpoint or points. The system then searches for other points having red,green, and blue values within a predetermined variance. The system thendefines a boundary that separates qualified points from the rest, andvectorizes this boundary to obtain a closed polygon. This polygondefines a user defined window as in the incorporated patent.

The operator has the capability to adjust the initial mean value and thevariance for each of the red, green, and blue channels. This allowsinclusion of more or fewer points from the collection of qualifyingpoints, and a new polygon or user-defined window can be defined.

In operation, the operator selects at least one region of a startingimage frame. The operator also selects an ending image frame. The systemcan then be commanded to search for similarly colored regions in theintervening frames and create user-defined windows for each interveningframe by applying the selection criteria (for example, within apredetermined variance of the mean red, green, and blue). This allowscreation of a new, different user defined window for each frame of thescene, automatically, which tracks the selected color region of interestregardless of geometric shape or location in the image.

According to another aspect of the invention, the operator can defineplural regions of interest in each frame, with each region havingdifferent selection criteria. The regions need not have the samecharacteristics, such as color or shape.

After the regions of interest are isolated and vectorized to obtain userdefined windows, the characteristics of the user defined windows can beedited to allow the user to adapt the window to the outline of theregion of interest. Modification of a user defined window comprisesadjusting the number of points representing a window, and adjusting thelocation of one or more selected points of a window.

Advantageously for use in a real time processing environment such as ascene-by-scene color corrector, the operation of automatically trackingthe color region of interest from during playback of the scene iscarried out at a real time rate. Thus, the user can adjust aspects ofthe image processing for exemplary frames of a scene and observe theeffects of the image processing during playback of the scene on a realtime basis.

These and other features and advantages of the present invention may bemore clearly understood and appreciated from a review of the followingdetailed description of the disclosed embodiment and by reference to theappended drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block schematic diagram of a system for creating userdefinable windows for applying image processing, constructed inaccordance with the preferred embodiment of the present invention, shownin connection with a scene by scene color correction system.

FIG. 2 is a flow chart illustrating the general process of capturing akey frame, selecting a color region of interest, deriving a user definedwindow, and applying image processing in accordance with the userdefined window.

FIG. 3, consisting of FIGS. 3A through 3F, is an illustrative example ofthe general method carried out in the present invention of capturing akey, selecting a color region of interest, vectorizing to define awindow, and devectorizing the window to derive a video matte.

FIG. 4 is an illustrative example of the general method of defining auser definable window associated with a region of interest in a firstframe of a scene and autotracking the region of interest to a finalframe of the scene.

FIG. 5 is an illustrative example of the employment of a user definedwindow from a prior frame to provide the starting point for a search forthe color region of interest in a subsequent frame.

FIG. 6 illustrates the seed point of a region of interest.

FIG. 7 is a state diagram illustrating various modes of operation of thepreferred system.

FIG. 8 illustrates an image capture control window displayed in thekeystore mode.

FIG. 9 illustrates a color region of interest (ROI) mode view/editwindow.

FIG. 10 is a flow diagram of relevant aspects of the preferredworkstation software that carries out the computer-implemented processemployed in the present invention.

FIG. 11, consisting of FIGS. 11A and 11B, is a flow diagram of the colorregion of interest computer-implemented process.

FIG. 12 is a flow diagram of a color grabber computer-implementedprocess.

FIG. 13 is a flow diagram of a first frame computer-implemented process.

FIG. 14 is a flow diagram of an ending frame computer-implementedprocess.

FIG. 15 is a flow diagram of an autotrack computer-implemented process.

FIG. 16 is a flow diagram of a variance adjust computer-implementedprocess.

FIG. 17 is a flow diagram of an add region computer-implemented process.

FIG. 18 is a flow diagram of a detect computer-implemented process.

DETAILED DESCRIPTION OF THE DISCLOSED EMBODIMENT

Referring now to the drawings, in which like numerals indicate likeelements or steps throughout the several figures, FIG. 1 illustrates animage processing system 10 constructed in accordance with the preferredembodiment of the present invention, suitable for carrying out theclaimed methods for defining windows for applying image processing andautotracking of color regions of interest.

Preferred Embodiment of Color Correction System

Although the present invention will be described in the context of ascene by scene digital color corrector, it will be understood that theinventions described herein are not limited to video color correctionand are equally useful in other types of systems where there is a needfor application of image processing such as image compression, imagefiltering, noise suppression, animation, and any other type ofprocessing. In particular, the invention is useful in any imageapplication where there is a need to define a region of an image forreceiving one type of data processing and another region for receiving adifferent type of data processing. However, in the context of thedisclosed embodiment, the inventions are particularly useful in a sceneby scene color corrector where a plurality of frames comprising a scenemust be viewed and adjusted to create or modify a video program such asa movie, commercial, television program, etc.

As used herein, the term “key” is used to describe a predeterminedregion of an image that is used to define or derive a user definedwindow which is stored in a vector format, as well as the video matteproduced by devectorizing a user defined window which is employed toselectively apply image processing to image regions in accordance withthe matte.

In FIG. 1, an image processing system 10 constructed in accordance withthe preferred embodiment comprises several basic components—acomputer-based workstation 12 including monitor 15 and keyboard 20, acolorist's control panel 25, a digital color correction system 30, asource of images 32, a video display 40, and a destination for correctedimages 42. The workstation 12 preferably includes pointing devices suchas a mouse 18 and/or graphics tablet 19. The workstation 12 is coupledfor data communications via a network connection 22, preferably a highspeed network such as Ethernet, to the color correction system controlpanel 25 and color correction system 30.

The source of images 32 may be a telecine, video tape player, or adigital video storage medium. The source is connected to an Inputcircuit board 31 in the color correction system 30. It will beunderstood that the image signals internal to the color correctionsystem 30 are digital, and are stored and manipulated in a digital videoformat such as the International Radio Consultant Committee (CCIR)promulgated Recommendation No. 601-1 which defines a standard set ofdigitized color signals. CCIR Recommendation 601-1 (1986) is herebyincorporated by reference. It is known in the art to take CCIR 601-1digitized signal streams, convert same to corresponding RGB signals (inanalog or digital domains), and then to perform the primary andsecondary color corrections and modifications on the resultant signals.The corrected or modified signal can then be passed through a signalmatrix and reconverted to a digital bit stream through use of an analogto digital converter. The standard signals defined in Recommendation601-1 essentially consist of a luminance signal Y and two colordifference signals (R-Y) and (B-Y). It is well known that, since theluminance signal contains information on levels of red, green and blue(RGB), the three standard signals can be used to reproduce the RGBlevels for any given set of samples.

An Output circuit board 35 is also provided as a part of the colorcorrection system 30, and typically converts the digital signals withinthe color correction system back into analog form for display. Thus, theOutput circuit 35 is coupled via a video signal cables 37 to a videomonitor 40, to a utilization device 42 such as a video tape player orother image storage device, and to the workstation 12 so that theresults of image processing (e.g. color correction) can be viewed by theoperator on the workstation monitor. It will also be understood that thecorrected digital signals can be stored in digital form in a digitaldata storage device coupled to the system.

The preferred workstation 12 is an INDY™ model workstation manufacturedby Silicon Graphics, Inc. (SGI). The preferred workstation includes anR4600 RISC processor operating in excess of 133 MHz, internal 24-bit XLcolor graphics (with RGB single or double buffer color indexes), adigital video input port capable of storing 640×480×30 fps to memory, 2serial and 1 parallel data port, an Ethernet network port, a fast SCSI-2data port, at least 32 megabytes of RAM, and a 1.0 or larger Gb harddisk. Optionally, a different SGI model workstation could be employed,e.g. one including a video adapter card that allows support and use ofmultiple monitors. Further details of the preferred workstation areavailable in the literature supplied by the manufacturer.

The preferred workstation 12 includes on-board display driver circuit,namely, the 24-bit XL color graphics circuitry with RGB double buffercolor indexes, which provides video signals to at least one workstationmonitor 15. A separate, second video monitor 40 is utilized in thepreferred embodiment to display a target image, that is, a full screenrepresentation of the video image being processed in the system. If aworkstation model is employed that supports multiple monitors, thesecond monitor 40 may be connected to the workstation 12.

The preferred graphics tablet 19, which is particularly useful forcertain manipulations of key images and vectors as will be described, isa Wacom ArtPad™ or ArtZ™ graphics tablet manufactured by WacomTechnology Corporation, Vancouver, Wash. 98661.

The preferred color correction system 30 is a Da Vinci RENAISSANCE8:8:8™ digital color corrector manufactured by Da Vinci Systems, Inc.,Ft. Lauderdale, Fla. This color correction system includes variouscircuit board components that are employed to carry out digital videosignal processing, and is coupled to the control panel 25. The preferredRENAISSANCE 8:8:8™ control panel 25 comprises three separate panels eachhaving a combination of dedicated keys, soft keys, and soft knobs thatprovides many simultaneous functions utilized in the color correctionprocess. A dedicated keyboard 50 contains both dedicated and soft keysfor system feature control, display/list management, source anddestination motion control, editing, special effects features, and thelike. This keyboard also contains a slide-out alphanumeric keyboard (notshown) to enter names, comments, and other important sessioninformation. Included on this control panel is a “User DefinableWindows” (UDW) button or key 51 that is utilized to activate thefunctions described herein.

It will be understood that the preferred RENAISSANCE 8:8:8™ systemincludes software that is operative for controlling the image source anddestination, and reading and storing frame numbers in association withuser-defined scene identifiers or names. Thus, the system is able toprovide displays of scene lists that facilitate the user in associatingscene identifiers or names with exemplary images from the scene.

The control panel 25 further includes a joyball panel 53, which includesknobs, a trackball 54 for manipulating a cursor, keys for source setup,primary color balance, color grabbing, trackball control, and othereffects control. When the windows feature of the present invention isactivated, movement of the trackball 54 causes generation of a cursor 60that is displayed on the video monitor 40. The cursor 60 is utilized tomanipulate and select portions of video images displayed on the monitor,as will be described.

Finally, the control panel 25 includes a soft panel 57, which containsknobs and softkeys for dedicated 6-color secondary enhancement, specialeffects control, dedicated memory keys, and other functions not forminga part of the present invention. It will be understood that theprovision of primary and secondary color correction, and sets of primaryand secondary color correction, one for regions inside a user definedwindow and one for regions outside a user defined window, can be set andstored in memory by manipulation of controls on the soft panel 57.

The color correction system 30 includes several circuit boards thatprovide the functions required to carry out color correction or othertypes of image processing in the digital domain, and to control inputfrom the image source 32 and utilization or storage in the utilizationdevice 42. These circuits include a real time controller (RTC) circuit70 with network interface that provides a data communication networkinterface to the data communication network 22 coupling the colorcorrection system 30 to the workstation 12 and color correction systemcontrol panel 25.

The real time controller (RTC) 70 includes a dedicated microprocessorfor handling real time events. Real time events occur at the rate of oneevery 30 millisecond (ms), so as to maintain an event rate correspondingto at least a 33 frames per second (fps) display rate. Data comprisingan events list is provided to the RTC 70 via the data communicationsnetwork 22; this events list is provided to control various functions ofcomponents in the system, for example providing a set of colorcorrections for each frame of a specified scene being processed. Inparticular, the events list as employed in the present inventionincludes a data structure that identifies a particular frame number andscene number, information that specifies one or more user definedwindows for the particular frame, information identifying colorcorrections to be applied to areas inside the one or more windows of theparticular frame, and information identifying color corrections to beapplied outside the window of the particular frame, and otherinformation.

The events list in the preferred embodiment is a data structure havingframe numbers as a time reference. The data structure comprises asequence of data blocks, each block being related to a beginning framenumber and an ending frame number. Each block is called a “scene” whichcorresponds to a continuous segment of film or video shots. Each blockalso contains a set of color correction parameters and windowparameters. The color correction parameters include known parameterssuch as red gain, red gamma, red black, etc., for each RGB colorchannel. The window parameters include window shape, window size, windowlocation, and other information.

A user definable windows (UDW) generator board 75 constructed asdescribed herein includes a dedicated high speed microprocessor thatconverts vectorized user defined windows into a video matte or key on areal time basis at the real time event rate. The RTC circuit 70 providesthe vector representations of the windows to the UDW board 75 at thereal time event rate.

A key frame buffer (KFB) circuit board 78 stores a video image from theimage source 32, or from any other digital storage system, in a highspeed memory. This image is then utilized in accordance with methodsdescribed herein to derive a vector representation of any user definedwindows. Keys for vectorizing may be obtained directly from the imagesource 32 or be retrieved from prestored images stored in the memory ofthe workstation 12.

Color correction circuit 80 carries out digital color correction andother video parameter adjustment functions associated with the colorcorrection system 30. The block in FIG. 1 identified as the colorcorrection circuit 80 includes all of the remaining functions of thepreferred RENAISSANCE 8:8:8™ digital color correction system 30, andwill not be discussed further. Details of the operations and functionsof the preferred color correction system 30 are available in theliterature supplied by the manufacturer.

As will be understood by those skilled in the art, the preferred colorcorrection system 30 is operative to receive video images from an imagesource such as telecine 32, digitize the video images, capture adigitized image in the KFB board 78, display the captured image on thevideo monitor 40, receive color corrections entered by the colorist oroperator via the control panel 25, impose such color corrections uponvideo images as they are transferred through the system, and providedcorrected output signals via the output board 35 which are employed orstored in a utilization device 42 and/or simultaneously displayed on thevideo monitor 40.

In carrying out the operations for applying image processing to userdefined windows, a system operator or user interacts with the systemthrough the control panel 25, and via the user interface elements of theworkstation 12, namely, keyboard 20, mouse 18, graphics tablet 19, andworkstation monitor 15. In particular, the workstation 12 executescomputer programs for carrying out the computer-implemented methodsdescribed herein, and generates displays on the monitor 15 containingwindows with user interface elements such as informational displays,control buttons, slider bars, data entry fields, image display areas,etc.

The reader should be careful here to distinguish user-interface“windows” displayed on the monitor 15 generated from execution ofcomputer software for carrying out the invention, from the “user definedwindows” for applying image processing. During system use, theworkstation monitor 15 will typically be displaying one or moreuser-interface oriented windows for carrying out the methods asdescribed herein. One such user-interface window is a control window 44,which comprises a display of various buttons and slider controls formanipulation of the key image. Activation of certain commands causes thedisplay of other, secondary windows.

One of the secondary windows is a View/Edit Window 46, shown overlappingthe control window 44 in FIG. 1. The View/Edit Window 46 is used todisplay a captured image in greater detail, and to provide certainadditional user controls specifically associated with editing of theimage, identifying regions of interest, etc.

The system operator manipulates controls in the Control Window 44 (ofwhich there are several types), and interacts with the images, vectorwindows or keys, etc. through various View/Edit windows 46 of differenttypes.

It should be noted that the video images displayed in the video monitor40 are preferably displayed within a particular View/Edit window 46.Since the images provided from the image source are digitized forprocessing in the preferred RENAISSANCE 8:8:8™ processing system, thesystem readily generates digitized video images and provides them in anappropriate format, such as Apple Computer's Quicktime™ video format,MPEG I or II, or other digital video format via the network 22 to theworkstation 12. Accordingly, it will be appreciated that any displaysdescribed herein where a video image is shown on the video monitor 40are typically also displayed (albeit in reduced size) on the workstationmonitor 15. Furthermore, any of the functions described in conjunctionwith a video monitor including generation of a cursor of a display ofoverlays of user defined window over a video image, display of a key,etc. can also be supplied in a window in the workstation display 15.

Thus, it is to be expected that as more and more image processing offilm and video is conducted in the digital domain, as opposed to thetraditional analog domain, methods of the present invention will beutilized in strictly digital systems for image storage, retrieval,manipulation, display, and output.

In the disclosed embodiment, however, the results of image processingare viewed on the video monitor 40, while the user interface andcontrols aspects of the image processing and the invention are carriedout through the workstation 12 and its associated monitor 15.

Overview of Method for Color Region of Interest

As partly described in the referenced patent, the system operates in aplurality of different primary modes. One is a keystore mode that allowscapture, editing and manipulation of 10-bit and 1-bit key images basedon luminance capture, another is a vector editing mode that allowsmanipulation of user-defined vectorized windows dry from key images inthe key store mode, and another is a color region of interest (ROI) modethat allows definition of regions by color information and manipulationof the vectorized regions of interest.

Turn next to FIG. 2 for an overview of the general process 100 ofcapturing a color keyframe, selecting a color region of interest,editing the key frame to derive a key, deriving a user-defined windowfrom the key, causing the color region of interest to autotrack theselected region of interest over a plurality of frames in a scene, andapply image processing in accordance with the user defined window. Itwill be understood that the method described in FIG. 2 is supplementaryand parallel to that described in connection with the referenced patent.

It should be understood that in the following several discussions ofmethods and processes involved in the present invention, numerousdetails are provided such as computer display system elements, aspectsof computer program user interfaces, display screens and formats, sampledata, etc. in order to provide an understanding of the invention.However, those skilled in the art will understand that the presentinvention may be practiced without the specific details. Well-knowncircuits, programming methodologies, and structures are utilized in thepresent invention but are not described in detail in order not toobscure the present invention.

Certain of the method descriptions which follow are presented in termsof exemplary display images, algorithms, and symbolic representations ofoperations of data within the computer's memory. As will be known tothose skilled in the programming arts, and particularly those skilled inobject-oriented programming methodologies, these algorithmicdescriptions and graphic exemplary displays are the means used by thoseskilled in the art of computer programming and computer construction toconvey teachings and discoveries to others skilled in the art.

For purposes of this discussion, an “algorithm” is generally a sequenceof computer-executed steps leading to a desired result. These stepsrequire physical manipulations of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical, magnetic,or optical signals that are capable of being stored, transferred,combined, compared, or otherwise manipulated. It is conventional forthose skilled in the art to refer to these signals as bits, values,elements, symbols, characters, images, terms, numbers, or the like. Itshould be kept in mind, however, that these and similar terms should beassociated with appropriate physical quantities inside the computer andthat these are merely convenient labels applied to these physicalquantities that exist within the computer.

It should also be understood that manipulations within the computer areoften referred to in terms such as adding, comparing, moving, etc. whichare often associated with mental operations performed by a humanoperator. It must be understood that no involvement of a human operatoris necessary or even desirable in many aspects of the present invention,since the operations described herein (except those specificallyidentified as origination with the operator or user) are machineoperations performed in conjunction with a human operator or user thatinteracts with the computer. The machines used for performing theoperation of the present invention, as will be understood, includegeneral purpose digital computers or other similar computing devices.

Furthermore, it should be kept in mind that there is a distinctionbetween the methods, steps, or operations carried out by a computer, andthe method of computation itself. The present invention does not involvea method of computation. The present invention rather relates tomethods, steps, or operations for a computer and processing electricalor other physical signals to generate desired physical signals anddisplay results and interactions. As illustrated in FIG. 1, the presentinvention also relates to a system 10 for performing these operations.Furthermore, it should be understood that the programs, algorithms, etc.described herein are not related or limited to any particular computeror apparatus. Rather, various types of general purpose machines may beused with programs constructed in accordance with the teachings herein,and employed as the workstation 12. Similarly, it may prove advantageousto construct specialized apparatus to perform the method steps describedherein by way of dedicated computer systems with hard-wired logic orprograms stored in nonvolatile memory such as read only memory.

It is assumed in FIG. 2 that the user has manipulated an image sourcesuch as 32 in FIG. 1 and has advanced to a location on a film orvideotape to view an exemplary image on a video monitor 40 that isdesired for use as a key image. Typically, this will be the first imageor last image of a scene, although this is not required and any imagecould be used.

At step 101, the user enters a command to capture a key frame from theimage source; this key frame is captured and stored in digital form inthe KFB circuit 78. A key frame at this stage in the disclosedembodiment for color region of interest tracking is a 24-bit colorimage, with 8 bits each of red (R), green (G), and blue (B) videochannels. The key frame image, a color image in the disclosedembodiment, is displayed on the video monitor 40 as it is viewed fromthe image source; after capture it is displayed in a miniaturizedversion in an image area in a control window 44 on the workstationmonitor.

At step 102, the system determines if a user defined window (UDW) from apreceding frame is available. In the case when the operator is firstselecting a color region of interest, there will not be a prior userdefined window in existence. On the other hand, after the operator hasselected a color region of interest for a given frame, a user definedwindow associated with this color region of interest will be inexistence and stored in the system memory. This prior-frame user definedwindow is employed as a starting window for a subsequent frame. Inaccordance with methods described herein, the beginning user-definedwindow from a prior frame is used to define a search area wherein thesystem begins to search for pixel color values within the predeterminedvariance of the mean color established for that prior user definedwindow.

If no user defined window from a preceding frame is available, then atstep 104, the user employs a tool to select a point or points in thecolor region of interest. As will be described, this feature isactivated by clicking on a “color grabber” button that causes display ofa predetermined cursor.

At step 106, the system computes the mean of the points selected by theoperator and displays this mean as a color in a predetermined displayregion on the display screen 46.

Step 110 shows that the operator can adjust the mean and the variance ofthe color value so as to provide an adjusted mean.

Using the adjusted mean and variance, at step 112 the systemautomatically determines the color boundaries of the region of interestby conducting a search algorithm to isolate the x and y coordinates ofpicture elements that meet the selected criteria of being within apredetermined variance of the mean color value.

Then, at step 118, this region, identified as a color region ofinterest, is vectorized to obtain a user-defined window (UDW). At step120, this UDW can be edited and at step 114 stored and retrieved.

At step 118, the isolated key image, which is also called a “key”, isvectorized at the workstation to obtain a user defined window (UDW)defined in terms of points, with vectors extending between the points.The term “vectorized” means that a continuous polygon of n points hasbeen drawn around the region of interest, to obtain a set of pointswhich can be connected with (and displayed as) a set of lines. In thisdisclosed embodiment, UDW's are defined in terms of points within animage space having 586 horizontal lines, with 720 pixel locations perline, where the last point of a set of points defining a window connectsto the first point.

After the key image has been vectorized at step 118 to obtain a vectorrepresentation of a window, the window can be stored in theworkstation's memory as indicated at the optional step 114. Further, aprestored vector window can be retrieved from memory, edited, andre-stored if desired.

At step 120, the user may edit the UDW if necessary or desired. The UDWediting process essentially entails operations of adding, deleting,and/or moving a point, a selected group of points, or an entire window.Windows for editing can be derived directly from the vectorizing step at118, by free hand drawing with a pointing device (e.g. mouse, graphicsor digitizer tablet), or by retrieving a prestored window from storagein the workstation 12, as indicated at 114. Edited windows can also bestored in the workstation's memory.

After the user has created a suitable UDW, it is used to apply imageprocessing, as indicated generally at step 122. In the disclosed systemfor color correction, using the window to apply processing involvesdevectorizing the window for a frame of video to thereby obtain adigital key, and employing the key as a mask or matte for applying colorcorrections. Regions corresponding to one or more windows receive apredetermined set of color corrections, while other regions receiveanother set of color correction.

For use in a series of frames of a scene, the user associates a firstuser defined window with a first frame of a scene, typically edits thefirst frame's window or defines a new window for the final frame of thescene, and then designates the windows as corresponding to the beginningand ending frames of the scene. The preferred system is thenautomatically operative to employ the first user defined window andautomatically track the color region of interest there between duringthe succession of frames in the scene in order to provide aprogressively changing window throughout the frames of the scene. Thus,step 124 in the disclosed embodiment involves applying color correctionon a frame by frame basis to regions of each frame in accordance withone or more user defined windows for each frame.

Other types of image processing that can be applied in accordance withuser defined windows include image compression, image filtering,animation creation, noise reduction, and colorization. Other types ofimage processing applications will occur to those skilled in the art.

Example of Color Region of Interest

FIG. 3 provides a specific example of these steps in operation. In thisexample, an image containing a bottle 150 and a glass 155 on a tablecloth is employed as the keyframe, and a key corresponding to only thebottle is derived. In particular, the key comprises the predeterminedcolored region of the bottle, and also the region 157 (FIG. 3D)comprising the top of the glass, which for purposes of the discussionexample, are assumed to comprise a predetermined due, with all the pixelvalues in the color region of interest being within the predeterminedvariance of the mean color value selected by the operator.

Referring first to FIG. 3A, an original color image is captured anddisplayed on the video monitor 40 (not shown). This image is alsodigitized and displayed in a view/edit window 46 on the workstationmonitor. Thus, the video monitor 40 will display the color image, andthe view/edit window 46 will display a digitized version of the imagefor manipulation by the operator. In starting, the user typicallyselects the key store mode of operation.

In FIG. 3B, the user has selected the “color ROI” mode of operation andactivated the color grabber control, thereby causing display of a “colorgrabber” icon 153. The operator uses this icon and clicks with the mouseor drags to select one or more points that are employed to determine themean color of a region of interest. The system automatically calculatesthe mean and displays it, and the operator can adjust the mean andvariance.

In FIG. 3C, either automatically after a point selection operation orupon provision of a predetermined “detect” command, the color region ofinterest is automatically vectorized. This results in the display of avectorized region 156 comprising a continuous polygon overlaid on top ofthe color image.

In FIG. 3D, the operator can select an “edit” mode wherein the points ofthe vectorized region 156 are adjusted in location, the mean andvariance can be adjusted, or a new region such as that shown at 157 canbe added by invoking a predetermined “add region” command.

After the vectorized regions have been modified to the operator'ssatisfaction, the user

-   -   can select the 1-bit mode, which effectively completely the        conversion of the image into a black and white image, with black        regions defining the area of interest that will ultimately        define the window for applying image processing.

In FIG. 3E, only the vectorized key windows are displayed as vectorregions 160. In FIG. 3F, the conversion to the 1-bit key image iscomplete, leaving two separate key areas or processing regions 165 ofthe image corresponding to the bottle and the glass.

The vector window(s) 160 are displayed on the workstation monitor, andare preferably overlaid on the key image so the user can visually verifythe accuracy of the user-defined window relative to the key image. Inthe example shown, the vector window comprises a plurality of pointsconnected by piece-wise linear segments. The user may select more orfewer points, as desired, so that the vector window matches the area ofinterest as closely as possible. The vector window may be edited byadding or deleting points, as desired.

In operation of the system to define windows for a plurality of framesof a scene, the user will define a window for the first frame of thescene and the last frame of the scene, as will be described.

The vectorized user defined window(s) 160 are utilized in the UDWcircuit board to create a digital matte that is electronically mixedwith the original image to define a region for applying colorcorrection. Stated in other words, the open regions 165 in FIG. 3Freceive a predetermined set of color corrections, while the remaining,black region 162 receives zero color correction, utilizing thewell-known video key mixing techniques known to those skilled in theart.

In addition, the present invention provides for a softness filter region170, which is shown as a white outline around the processing region 165.As described in the referenced patent, color corrections are applied inthe softness filter region 170 gradually tapering from zero correctionat the boundary adjacent the non-processed region 162 (zero percent) to100 percent processing at the boundary of the processing region 165. Thewidth or gradient of the softness filter region 170 is selectivelyadjustable by the user.

With the foregoing in mind about the method for creating a user definedwindow associated with a predetermined color region of interest, turnnext to FIG. 4 for an illustration of the application of the method ofdetermining a predefined user defined window associated with a colorregion of interest in a first frame of a scene and in the final frame ofthe scene and the manner in which the system automatically tracks thecolor region of interest through a plurality of scenes in a frame.

Using the example of the bottle and glass, assume that frame 1 of ascene having n frames shows the bottle 150 and glass 155 in an initialposition of the screen, as displayed on the video monitor 40 and in theview/edit window 46. As the scene progresses through its frames untilthe final frame n the bottle is gradually tilted forward, the corkremoved, moved in position toward the upper center of the screen fromthe left of the screen, and changed in orientation from a side view ofthe bottle to a top view, as the bottle is turned towards the view, withthe contents being poured into the glass. Thus, the bottle changes itsposition, size, and geometry (shape) from frame one to frame n. However,the mean color of the bottle does not change as the bottle changes itsposition, size, and geometry.

With the present invention, the user-defined window 160 effectivelytransforms its shape from the elongate bottle to a circle throughout then frames. The system is automatically operative to detect the mean colorin the region of interest defined by the starting color region key 160 adetect that predetermined color in the successive frames 2, 3, . . . netc. and to vectorize the color region of interest so as to define theregions for applying image processing. At any point along the way, theoperator may freehand draw a window, derive a new key from the videoimage of a frame, add a new key or region of interest, use a prestoredkey such as a circle or square, or may use a prestored window derivedfrom a prior frame and edit the characteristics to define a region suchas that shown at 160 n.

In order to effect an orderly transition between geometry over a numberof frames, the user may want to maintain a correspondence between thepoints of the starting window 160 a and the ending window 160 n.Typically, this correspondence is maintained by providing the samenumber of points in the ending key window or vector 160 n as in thestarting key window or vector 160 a. The number of points is selectablyadjustable by the operator.

In the referenced patent, there is a mapping between predeterminedpoints and a starting and ending key window. In the present invention,there is no need to maintain a correspondence between particular pointsin the key windows, and the key windows are employed merely to definethe color region of interest.

A preferred approach when a user-defined window changes shape, size, andposition is for the operator to work backwards from a window for a finalframe and edit the window of the final frame to derive a window for thefirst frame by adjusting the color parameters so that the mean hue andvariance from the mean effectively capture the region of interestthroughout the plurality of frames. In this approach, the operator viewsthe last frame of a scene and determines the number of points requiredto represent the user-defined window for the region of interest. Thisnumber of points is then noted as a minimum number of points for use asthe starting window. The user then views the window for frame 1 andadjusts the color parameters as appropriate.

It should be understood at this juncture that the number of framesdefining any given “scene” is completely up to the discretion of theoperator. If the transformation of the region of interest between astarting window and an ending window is not acceptable, the user maydefine a subscene comprising a smaller number of frames that provides amore satisfactory transition. The capability of assigning any number offrames to a scene is provided in the preferred RENAISSANCE 8:8:8™ colorcorrection system.

Of course, the ultimate worse case operation is to create and assign auser defined window to each frame individually, but in most cases thiswill not be necessary. The present invention allows storage of aplurality of user defined windows, each having a number of points,limited only by the total number of points defining such windows foreach frame of a program. As described in greater detail below, the totalnumber of points in the user defined windows is limited to a number thatallows the UDW board 75 to devectorize all windows and create thedigital matte on a real time basis, which is 200 in the disclosedembodiment.

UDW Generator Circuit

The user definable window (UDW) generator circuit 75 employed in thepreferred system shown in FIG. 1 is described in detail in thereferenced patent. Further details will not be provided herein. Asdescribed, the UDW board is operative to receive a command stream fromthe RTC circuit 70 comprising vector window definitions, and convertthese vectorized windows into a digital mattes or keys that can beemployed by other digital signal processing circuitry to mix with thedigital video signal to selectively apply image processing such as colorcorrection with the predefined regions of interest.

The softness filter 300 employed in the UDW circuit 75 is also describedin the referenced patent, and will not be described further herein.

FIG. 5 is an illustrative example of the method carried out in thepresent invention whereby a user-defined window associated with aselected color region of interest for a first frame is employed in asecond frame to facilitate definition of a subsequent region of interestand user-defined window in the subsequent frame. The present inventionoperates under the assumption that a particular color region of interestbetween two immediately consecutive frames will not change significantlyin size, position and shape between such consecutive frames.Accordingly, the user-defined window from a prior frame is employed as abeginning user-defined window for a subsequent frame; steps are taken toemploy this user-defined window as a search region to begin a search forthe color region of interest.

Assume in FIG. 5 that the color region of interest is defined by thebottle 150. In the second diagram, the selected region of interest isshown with its associated vectorized user-defined window 160. In frame 2of the scene, note that the position and orientation of the bottle haschanged to a tilted position. In the present invention, the vectorregion 160 from the prior frame (frame 1) is utilized as a startingvector region of interest for a subsequent frame (frame 2). Thisstarting vector region is employed to define a region of search forpixels having a color value within the variance of the mean employed todefine the color region of interest. Preferably, the search begins at apredetermined “seed point” 172, such as the calculated center of gravityof the polygon defining the vectorized region 160. Starting with thepixel associated with the seed point, the system searches the pixels ina predetermined manner so as to identify pixels having a color valuewithin the predetermined variance of the mean. After the search andredraw of the polygon, the new vectorized region of interest 160′ isdrawn.

FIG. 6 illustrates a search from the color boundaries beginning at theseed point 172 so as to determine the region of interest defined by thetilted bottle 150′. In the preferred embodiment, the seed point 172 iscalculated by determining the center of gravity of the vectorizeduser-defined window such as that shown at 160. Those skilled in the artwill understand how to compute and determine the center of gravity of ageometric polygon based on well-known programming techniques.

Furthermore, in the present invention, a bounding rectangle 174 isdefined that encloses the entire polygon 160 and defines an outer searchlimit.

The searching methodology is as follows: beginning at the seed point 172of the vectorized region of interest 160 from the preceding frame, theRGB values of the pixel at the seed point are examined to determinewhether or not they satisfy the criteria of being within thepredetermined variance of the mean value of the color of interest. Inmost cases, this will result in a “yes” determination. The system beginsby searching first to the left of the seed point until it encounters apredetermined color boundary defined by a predetermined color valueoutside of the mean and variance. This point is identified as aleft-most color transition edge. In like manner, the system searchesfrom the seed point to the right to determine a color transition edge,to the top, and then to the bottom to determine four color transitionedges. These four points then define initial boundaries for furthersearching.

Those skilled in the art will understand that a number of differentalgorithms for searching can be employed, including but not limited toan inverted spiral search and a simple progressive scan in x and y fromthe seed point.

Furthermore, the preferred algorithm will employ methods that terminatethe search in the event that the maximum geometric boundary such as thatof the bounding rectangle 174 are exceeded to change search methodologyto that of random searching or in the worse case a progressive scan ofevery value in the array.

Furthermore, the vectorized regions of interest and their respectivecenters of gravity can be employed to determined a trajectory, velocityand acceleration for the center of gravity. Such methodologies can beemployed to project the direction and rate of change of the region ofinterest so as to locate a seed point for searching of the next frame ata position within a subsequent frame corresponding to that of thevelocity and/or acceleration of the region of interest across one ormore frames.

User Interface of Software for Automatic Region of Interest Tracking

FIG. 7 is a state diagram illustrating the various modes of operation ofa system constructed in accordance with the preferred embodiment of thepresent invention. These are the principal modes of the software thatruns on the workstation 12 to provide the basic operations of capturinga key frame, isolating a region of interest based on color or on anotherparameter, vectorizing the region of interest to create a user definedwindow, editing the window(s) for association with particular imageframes in a scene, and automatically tracking the region of interestthrough a plurality of frames.

The software primarily executes in four basic modes—a keystore mode 360,a vector mode 365, an edit window mode 368, and a color region ofinterest (ROI) mode 369. Each mode results in the generation ofparticular user interface displays or windows (as contrasted with userdefined windows) on the workstation monitor 15—control windows andview/edit windows—with corresponding displays on the video monitor 40,which collectively allow user interaction with the system. Within theprincipal modes, certain types of operations are possible which areappropriate for the task at hand.

The three modes of operation of keystore mode 360, vector mode 365, andedit window mode 368 are described in the referenced patent and will notbe repeated herein. Thus, the discussion which follows is primarilyfocused on the color ROI mode 369.

Operation within the various modes results in display of various userinterface or control windows such as an image capture window 44 andview/edit windows 46 on the workstation monitor. FIG. 8, for example,illustrates a control screen 44 a that is displayed in the keystore mode370. The control screen 44 a is the initial control screen that comes upwhen the workstation software is initiated, but other types of controlwindows are displayed in other modes of operation.

In each control window 44, selection between the keystore mode, thevector mode, the edit window mode, or the color ROI mode is made bypressing (i.e. clicking with the mouse) a Key Store button 370, a Vectorbutton 375, an Edit Window button 376, or a Color ROI button 377.Selection of these buttons by the operator activates the correspondingmode.

Still referring to FIG. 8, a View button 378 turns on and off aview/edit window whose contents vary depending upon the mode. Thecontrol window 44 includes an image region 430 that displays a smallversion of the image, user defined window, etc. When the user selectsthe View button 378, a larger version of this image is displayed in aseparate view/edit window 46, of which there are different types. In thecase of the Color ROI button 377, a view/edit window 46 as shown in FIG.9 is displayed.

Other details of the buttons in the image capture control window 44 a inFIG. 8 are provided in the referenced patent.

Referring now to FIG. 9, the Color ROI view/edit window 46 is displayedin the color ROI mode. This window 46 includes a display region 410where the color image being operated upon is displayed, with an overlayof the key vector or user-defined window 160. In the example given, thepartially rotated bottle from prior figures is displayed, with avectorized key of the current region of interest 160 comprising a closedpolygon displayed superimposed over the color image.

In connection with automatic region of interest tracking, a number ofcontrols are provided. First, a color grabber button 412 when depressedcauses display of a “color grabber” cursor icon as shown at 153.

A mean color display region 415 provides a visual display of the averageor mean color as determined by the system, in response to selection of apoint or points by the operator.

A plurality of variance sliders 416, one for each of the red, green, andblue channels, is provided to adjust the variance from the mean. Theoperator can adjust the variance between a minimum of zero and a maximumof 256 possible values (an 8-bit variance is provided). A selectedvariance of 20 is shown in FIG. 9 for each of the three variancesliders. Adjustment of these variance sliders controls the variance fromthe mean that a given pixel color value can assume while still beingincluded within the color region of interest. If a color pixel valueassumes a value within the variance parameter of the mean, that pixelwill be included within the color region of interest and will beenclosed within the vectorized key 160.

A first frame control button 417 is provided to allow the operator toselect and/or rewind to the first frame of a scene for manipulation. Anending frame control button 418 is provided to allow the operator toselect and/or advance to the ending frame of a scene.

A play button 420 is provided for commanding the system to display theframes in the scene for viewing by the operator. A “start auto tracking”button 422 is provided to allow the operator to command the system tobegin with a region of interest defined in a first frame and track theparticular region of interest in successive frames, one frame at a time.

A plurality of points controls 424 are provided to allow operatoradjustment of the number of points (Points) in the closed polygoncomprising the vectorized key such as 160, and the minimum distancebetween points (Dist).

A Detect button 425 is provided so that the operator can command thesystem to initiate the process of calculating the mean and detect colorsin the image within the predetermined variance of the mean.

A Done button 426 is provided so that the operator can signal to thesystem that operation with this particular window in this mode arecompleted.

FIG. 9 also illustrates the provision of predetermined boxes or radiusaround a selected point for computing the mean color value. For example,a predetermined pixel (x₁, y₁) is shown within the region of the bottle150. A 3×3 pixel window or “box” is shown surrounding this pixel. Byselecting the “Define Box” button 433, the operator can command thesystem to define a box of predetermined size for use in computing themean. The system is automatically operative to compute the mean colorvalue of the pixels within the 3×3 box surrounding the selected pixel.The pixel is selected with a click operation.

Similarly, the pixel at location is shown in a 9×9 pixel window. Thesystem is automatically operative to computer the mean color value ofthe pixels within the 9×9 window.

Alternatively, the user can define a predetermined radius around aselected point, as shown at the point (x₃, y₃). The point (x₃, y₃) isshown within a portion of the glass 155, with a circle defined by afour-pixel radius around the point. If the operator selects the “DefineRadius” button 432, the system determines the average color value withina radius of n pixels measured from the center point (as defined by aclick).

For this purpose, a Define Radius button 432 is provided in the window46. Activating this button causes display of a predetermined dialog boxthat allows the operator to set the radius around a selected point. Thepoint is selected by a click operation when in the “Define Radius” mode.

A “Define Box” button 433 is provided so that the operator may define apredetermined square box comprising a predetermined number of pixelsaround a selected point as shown for the points (x₁, y₁) and (x₂, Y₂).

Finally, an Add Region button 434 is provided. Actuating this buttonallows the operator to “freeze” and save a particular vectorized keysuch as that shown at 160, and activate a mode wherein a new mean valuecan be selected so as to allow creation of another, second vectorizedkey for another region of interest in the same image.

Other tools, not forming a part of the present invention, are alsoprovided in the preferred system so as to allow other types of imagemanipulation, some of which is described in the referenced patent.

From the foregoing, those skilled in the art will appreciate that anumber of different tools are provided in a system constructed inaccordance with the present invention that allow the user to create andedit user defined windows—either windows created from an immediatelypreceding vectorizing operation carried out on a key image, or windowspreviously created during prior operations and stored in the memory ofthe workstation. After any desired window editing operations, the userdefined window (or windows, as plural windows are permitted in thepreferred embodiment) is associated with a selected frame of a scene,and the controls in the control screen 44 activated to cause imageprocessing to occur in accordance with the commands issued, e.g. applycolor correction.

Preferred Software Methods

FIG. 10 is a flow chart illustrating the steps of a computer-implementedprocess 1000 carried out in the work station 12 to carry out methods ofthe present invention described herein. These steps include steps forcapturing a keyframe, isolating a color region of interest for defininga key, deriving a user-defined window from the key, and applying imageprocessing in accordance with the user-defined window for a sequence offrames in a scene. The sequence of steps in this figure relates to amethod for deriving a key and corresponding window for a first frame ofa scene, deriving a key and corresponding window for subsequent frame ofa scene, automatically tracking the color region of interest over aselected number of frames in the scene, and applying image processingacross the frames in the scene. In particular, the reader is referred toFIGS. 3, 4 and 5 as illustrative examples of the method as carried out.

Starting at step 1001, the first step taken is to initialize the systemand load the software for the workstation 12 by allocating a region inmemory for handling the software processes and storing relevant data. Atstep 1003, the image capture window (or other suitable starting displaywindow) such as that shown in FIG. 7 is displayed.

At step 1004, the program enters an infinite loop of getting a usercommand or receiving an event that drives subsequent actions. Thoseskilled in the art will understand that the preferred software isevent-driven and actions occur as a result of a user typing a command,pointing with the cursor and clicking on an object (such as a control),clicking and dragging, etc.

At decision step 1005, the inquiry is made whether the Key Store buttonhas been depressed. If so, process 1020 is executed and the systementers the key store mode. Steps associated with the key store mode ofoperation are described in the referenced patent.

If the Key Store button has not been pressed, at decision step 1012 theinquiry is made whether the Vector button has been pressed by the user.If so, process 1030 is executed to enter the Vector mode. Stepsassociated with the vector mode of operation are described in thereferenced patent. If not, at decision 1035 the inquiry is made whetherthe Edit Window button has been pressed by the user. If so, process 1040is executed and the system enters the edit window mode. Steps associatedwith the Edit Window mode of operation are described in the referencedpatent.

If at step 1035 the Edit Window button has not been pressed, controlpasses to decision 1045, where the inquiry is made whether the Color ROIbutton has been pressed. If so, the process 1100 is executed. Stepsassociated with the Color ROI mode of operation are described inconnection with FIG. 11.

FIG. 11 illustrates the steps of the process 1100 associated with thecolor region of interest (ROI) mode of operation. The first step takenat step 1105 is to cause display of the color ROI window 46, a type ofview/edit window, as shown in FIG. 9. Control then passes to step 1107where the system enters an infinite loop to scan for user commands orevents. For purposes of the present invention of determining a colorregion of interest and autotracking the region of interest over aplurality of frames, several of the control buttons shown in FIG. 9 willbe actuated by the operator.

If at decision 1109 the Capture button 413 has been pressed (FIG. 9),step 1110 is carried out to capture and display an image from the imagesource.

If at decision 1112 the “color grabber” tool button 412 (FIG. 9) hasbeen pressed, process 1115 is executed to carry out a color grabberroutine. Steps associated with the color grabber routine are describedin connection with FIG. 12.

If at decision 1117 a first frame command has been provided bydepressing the First Frame control button 417, a First Frame routine iscarried out at 1119. steps associated with the First Frame routine aredescribed in connection with FIG. 13.

If at decision 1121 an Ending Frame command is provided by depressingthe Ending Frame control button 418 (FIG. 9), an Ending Frame routine1123 is executed. Steps associated with the Ending Frame routine aredescribed in connection with FIG. 14.

If at decision 1125 an Autotrack command is provided by depressing theStart Autotracking button 422 (FIG. 9), an Autotrack routine 1127 isexecuted. Steps associated with the Autotrack routine are described inconnection with FIG. 15.

If at decision 1130 a Variance Adjust command has been provided byactuation of one of the RGB variance sliders 416 (FIG. 9), a VarianceAdjust routine 1131 is executed. Steps associated with the varianceadjust routine are described in connection with FIG. 16.

If at decision 1133 the Play button 420 (FIG. 9), has been actuated,step 1135 is carried out to apply image processing to the predefinedregions of the scene. Although step 1135 is a process, those skilled inthe art will understand that the image processing as described in thereferenced patent are applied in the manner described therein. Suchimage processing is applied to each frame within the scene in real timeso that the operator can observe the effects of the isolation of theregion of interest and application of the image processing such as colorcorrection within the region of interest defined for each frame in thescene.

If at decision 1140 the Define Radius button 432 (FIG. 9) has beenactuated, step 1142 is carried out to display a dialog box to theoperator. Although the step 1142 is a separate process, it will beunderstood that this dialog box allows the user to enter a number as aradius so that a subsequent operation for computing the mean color iscarried out on all pixels within the predefined radius.

If at decision 1145 the Define Box button 433 (FIG. 9) has beenactuated, step 1147 is executed. Step 1147, a separate process, isoperative to display a dialog box that allows the operator to define thesize of a pixel window such as that shown in FIG. 9 as a 3×3 or a 9×9window. Other sizes of boxes are also possible. The system is operativein response to compute the color mean for all pixels surrounding a clickpoint as defined by the size of the window or box.

If at decision 1150 the Add Region button 434 (FIG. 9) has beenactuated, an Add Region routine 1152 is executed. In this routine, stepsare carried out for saving a particular user-defined window and enteringa mode wherein the operator can define and select a second user-definedwindow within the same frame. Steps associated with the add regionroutine are described in connection with FIG. 17.

At decision 1155, the inquiry is made whether the Detect button 425 hasbeen actuated. If so, a Detect routine 1157 is executed. The Detectroutine is operative to utilize the predetermined mean color value anddetect all pixels having a color value within the indicated variance ascontrolled by the RGB variance sliders and include those within an areawhich is then vectorized. Steps associated with the Detect routine aredescribed in connection with FIG. 18.

At decision 1160, the inquiry is made whether another command that isappropriate for the window 46 in FIG. 9 has been provided. If so, anappropriate command is executed as shown at step 1162. Finally, atdecision 1165, the inquiry is made whether the Done button 426 has beenactuated. If so, the “yes” branch is taken and the color region ofinterest routine exits and returns control to the calling processes. Ifnot, the program loops back to yet another user command or event at step1107.

FIG. 12 illustrates the Color Grabber routine 1115 that is carried outin response to activating the Color Grabber button 412 in FIG. 9. Thefirst step taken at step 1201 is to display the Color Grabber cursorsuch as that shown at 153 in FIG. 9. At decision 1203, the inquiry ismade as to whether the cursor is in the image area 410. If not, at step1205 the regular cursor is displayed, and the system is operative torespond to mouse clicks in the normal manner in regions outside thecolor image area.

If on the other hand the cursor is in the image area, the inquiry ismade at decision 1207 as to whether the mouse button is down. If not, noparticular action is taken and the program loops back to decision 1203.If the mouse is down, control passes to step 1209, since a potentialclick or drag operation has been indicated. The instantaneous value ofthe red, green, and blue color values at the location of the cursor asindicated by the variable (x_(n), y_(n)) is stored in an array for usein computing the mean color value. Control passes to decision 1212.

At decision 1212, the inquiry is made as to whether the mouse button hasbeen released (mouse up). If so, the “yes” branch is taken to step 1215,and the mean is computed for all of the values in the array that haveaccumulated since the mouse button was depressed. At step 1217, thecomputed mean is converted into a color and displayed within the meancolor region 415 on the view/edit window 46 (FIG. 9).

If at decision 1212 the mouse is still depressed, the “no” branch istaken to decision 1220. At 1220, the inquiry is made whether the cursorhas been moved. If not, control passes back to decision 1203. If so,control passes to step 1222 where an index (i) for tracking the possiblemovement of the cursor within the region of interest is incremented soas to add another value for purposes of computing the mean. Control thenpasses back to decision 1203.

The process 1115 continues to execute until the user presses the Donebutton, or actuates another command.

It will be understood that the steps shown in FIG. 12 are operative toemploy either a single click which results in a single value beingemployed as the mean color value, or to accumulate a number of colorvalues such as when the user drags the cursor within the image area soas to signify that a plurality of color values are to be employed indetermining the mean color for purposes of detecting the color region ofinterest.

FIG. 13 illustrates the First Frame Routine 1119 that is executed inresponse to the operator's actuation of the First Frame Control 417(FIG. 9). The first step taken at 1301 is to display the first frame ofthe current scene. At step 1305, the prestored vector key derived forthe first frame is retrieved from memory, and at step 1307, the vectorkey is displayed overlaid on the image in the View/Edit window 46 in thecolor ROI. If desired, the operator can edit the vectorized key byadding or deleting points, adjusting the distance, etc. using the pointcontrols 424 (FIG. 9).

FIG. 14 illustrates the Ending Frame routine 1123 that is activated inresponse to the operator's actuation of Ending Frame control button 418(FIG. 9). The steps of this routine are identical to that of the FirstFrame routine except that the actions apply to the ending frame of thescene.

FIG. 15 illustrates the Autotrack Routine 1127 that is executed inresponse to the operator's actuation of the Start Autotracking button422 (FIG. 9). The first step taken at 1501 is to retrieve the mean colorvalue and initialize a frame count index i=1. This initializes theroutine to begin autotracking at the initial or beginning frame beingviewed by the operator.

At step 1505, the frame count index i is incremented to advance to thenext frame. At step 1507, the vector key for frame i−1 is retrieved soas to obtain a starting key for beginning the search algorithm to locatethe color region of interest. At step 1508, the system displays frame iof the scene. At step 1511, the vector key for frame i−1 is overlaid onthe image for viewing by the operator. At step 1515, the systemdetermines the center of gravity for the vector key for frame i−1 todetermine the seed point.

At step 1520, the system begins at the seed point to execute analgorithm for searching for color values that are within thepredetermined variance of the mean color. This results in determinationof a region of pixels, not necessarily continuous or contiguous, thatcan be satisfactorily vectorized.

Then, at step 1525, the region of interest identified for all colorvalues meeting the criteria of values within the variance of the meancolor are vectorized. At step 1530, the vector key for the current framei is then stored. Control passes to decision 1535, where the inquiry ismade whether the final frame of the scene for autotracking has beenreached. If not, control passes back to step 1505 and the next frame isprocessed. The final frame has been reached, the “yes” branch is takenand the routine exits.

FIG. 16 illustrates the Variance Adjust routine 1131 that is executed inresponse to the operator's movement of the RGB variance sliders 416(FIG. 9). Starting at step 1601, the first step taken is to respond tomovement of the slider bars. It will be understood that because thedescribed system is an event-driven system, movement of any one of thethree slider bars for R, G, and B channels causes execution of thisroutine. Thus, the predetermined variance value associated with thechannel is then adjusted in accordance with the motion of the slider barassociated with the channel.

At step 1605, an inquiry is made as to whether the variance for theselected channel is at its maximum value. If so, at step 1607 themaximum variance value (256) is displayed on the screen and employed asthe maximum variance.

If at decision 1605 the variance is not at its maximum, the inquiry ismade at step 1609 whether the variance is at a minimum value. If so, theminimum variance (0) is displayed. If the variance is not at its minimumvalue, then control passes to step 1615, and the adjusted variance valueis displayed and employed as the selected variance value.

FIG. 17 illustrates the Add Region routine 1152 that is executed inresponse to actuation of the Add Region button 434 (FIG. 9). Starting atstep 1701, the first step taken is to inquire whether a region countindex j=0. The index j is employed to keep track of the number ofsimultaneous regions that are being employed in the current scene. Forexample, FIG. 3 illustrates employment of two vectorized key windowswith corresponding mattes. If the region count index is not at 0,control passes to step 1703. The region of interest vector, the mean,and the variance for the present region j is then saved in a memorylocation of the system. Control passes to step 1707, and the regioncount index j is incremented. At step 1711, the mean color for theregion is cleared so as to allow the operator to establish a new meancolor for identifying the region of interest. The routine then exits.

FIG. 18 illustrates the Detect routine 1157 that is executed in responseto the operator's actuation of the Detect button 425 (FIG. 9). The firststep taken at 1801 is to retrieve the mean color for the current regionj. At step 1802 the system searches for pixels in the image that havecolor values within the predetermined variance of the mean. At step1806, the identified pixels are employed as the basis for vectorizingoperation to establish a vector key for the region of interest. At step1808, the vector key is saved.

From the foregoing, it will be understood and appreciated that thepresent invention allows an operator to identify a region of interest byhue, saturation, and/or luminance, and particularly RGB colorinformation so as to identify a color region of interest, automaticallyestablish a vector key for the region of interest that can be selectablyadjusted so as create a region for applying image processing such ascolor correction and autotrack this region of interest based on the meancolor of the region of interest over a number of frames in a scene.

In view of the foregoing description of the preferred embodiments of thepresent invention, other embodiments will suggest themselves to thoseskilled in the art. Therefore, the scope of the present invention is tobe limited only by the claims set forth below.

1. In a scene by scene color corrector, a process for automaticallyapplying color correction to a selected region in plurality of frames ina scene, comprising: (a) in response to user commands, defining aselected region in a first frame of the plurality of frames; (b)generating a user defined window for the first frame corresponding tothe selected region; (c) for a subsequent frame of the plurality ofimages, generating a user defined window for the subsequent frame basedon a color parameter associated with a prior frame; (d) applying colorcorrection to areas within the user defined window region in thesubsequent frame.
 2. The process of claim 1, wherein the user definedwindow comprises a continuous polygon circumscribing the region.
 3. Theprocess of claim 1, wherein the steps (c) and (d) are repeated for eachframe of the plurality of frames.
 4. The process of claim 1, wherein thestep of generating a user defined window based on a color parameterassociated with a prior frame comprises the steps of: determining a seedpoint derived from the user defined window of the prior frame; startingat a point associated with the seed point, searching in the frameaccording to a predetermined methodology for picture elements that fallwithin a predetermined range of the color parameter; defining a boundarybased on a point in the frame at which the located picture elements falloutside of the predetermined range of the color parameter; andvectorizing the boundary to derive the user defined window.
 5. Theprocess of claim 4, wherein the predetermined range of the colorparameter comprises a user selectable variance.
 6. The process of claim1, wherein the color parameter comprises the mean of the color value ofa selected one or more picture elements.
 7. The process of claim 6,wherein the type of color parameter is selected from the groupcomprising hue, saturation, luminance, and RGB.
 8. The process of claim7, wherein the color value of the selected one or more picture elementscomprises the red (R), green (G), and blue (B) color values in an RGBcolor representation.
 9. In a scene by scene color corrector, a systemfor automatically applying color correction to a plurality of frames ina scene, comprising: a component that defines a region in a frame inaccordance with a color parameter of the frame; a component thatutilizes a region from a prior frame to define a region in a subsequentframe of the scene; and a component that applies color correction toareas within the regions of the frames over the selected plurality offrames in the scene.
 10. The system of claim 9, wherein the regioncomprises user defined window, the user defined window comprising acontinuous polygon circumscribing the region.
 11. The system of claim10, wherein the component that utilizes a region from a prior frame todefine a region in a subsequent frame comprises: a component fordetermining a seed point derived from the prior frame; a searchcomponent operative, starting at a point associated with the seed point,for searching in the subsequent frame according to a predeterminedmethodology for picture elements that fall within a predetermined rangeof the color parameter; a component for defining a boundary based on apoint in the subsequent frame at which the located picture elements falloutside of the predetermined range of the parameter; and a vectorizingcomponent for vectorizing the boundary to derive the region for thesubsequent frame.
 12. The system of claim 11, wherein the predeterminedrange of the color parameter comprises a user selectable variance. 13.The system of claim 9, wherein the color parameter comprises the mean ofthe color value of a selected one or more picture elements in the frame.14. The system of claim 13, wherein the type of color parameter isselected from a group comprising hue, saturation, luminance, and RGB.15. The system of claim 13, wherein the color value of the selected oneor more picture elements comprises the red (R), green (G), and blue (B)color values in an RGB color representation.